Predicting human behavior by machine learning of natural language interpretations

ABSTRACT

An accurate thought map is created by recording people&#39;s many utterances of natural language expressions together with the location at which each expression was made. The expressions are input into a Natural Language Understanding system including a semantic parser, and the resulting interpretations stored in a database with the geolocation of the speaker. Emotions, concepts, time, user identification, and other interesting information may also be detected and stored. Interpretations of related expressions may be linked in the database. The database may be indexed and filtered according to multiple aspects of interpretations such as geolocation ranges, time ranges or other criteria, and analyzed according to multiple algorithms. The analyzed results may be used to render map displays, determine effective locations for advertisements, preemptively fetch information for users of mobile devices, and predict the behavior of individuals and groups of people.

REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 62/438,731 filed on Dec. 23, 2016 by Pranav Singh, BernardMont-Reynaud, Kheng Khov, and Jonah Probell, and entitled GEOGRAPHICALMAPPING OF INTERPRETATIONS OF NATRUAL LANGUAGE EXPRESSIONS, which isincorporated herein by reference in its entirety.

FIELD OF THE INVENTION

The present invention is in the field of creating regional mapsindicating an aggregation of concepts included in thoughts of people atlocations illustrated by the map. Concepts may be accurately determinedby using natural language understanding techniques.

BACKGROUND

Maps that are two-dimensional pictures of a particular geographicallocation take many forms. Maps exist that indicate the activities ordemographics of people over mapped space. The information that is usedto represent an aggregation of activities and demographics of peoplehave largely come from public records and research studies such as crimestatistics, traffic patterns, residential social-economic studies,census data, market surveys, tax assessment data. It is relatively newfor such maps to mine data from the Internet contributed by individualusers. Furthermore, information previously available was not a reliablepredictor of future behavior by people in an area.

Data mining and machine learning by computers have been applied toeverything from setting credit scores to making investments topredicting crimes. Some systems use natural language processingaccording to widely used human languages, such as English, MandarinChinese, Japanese, and German, to extract meaning and intent from rawdata. Associating meanings with individual people, based on theirexpressions, may be an accurate predictor of their interests, futurebehavior, and even travel patterns. Mobile devices, and localizedterminals, allow systems to process natural language meaning and intentfor individuals across whatever locations they transit. However, suchsystems do not process natural language meanings and intents forparticular locations and for the various individuals that pass throughthe locations.

Some experimental systems have used associations between geolocationtags and key words in Twitter tweets or other social media messages tomake fascinating maps of sounds, smells, and emotions throughout cities.Such systems analyze data simplistically and provide data that is onlyuseful for simple applications. Such systems use specific sets ofkeywords identified by researchers. However, mining social media messagefor keywords may not provide an accurate indication of the user'sthoughts. Consider for example a user tweeting “the ball's in yourcourt” may identify sports-related concepts rather than negotiation. Thevalue of the thought map depends on the accuracy of identifyingconcepts.

Furthermore, keyword-based systems do not provide useful analysis acrossthe dimension of time. In particular, they do not recognize cyclicalpatterns in time, such as expressions at particular times of day or daysof week, and they do not recognize cyclical patterns in location, suchas from bus stop to bus stop or house to house.

SUMMARY OF THE INVENTION

The present disclosure relates to mapping concepts mined from utterancesof natural language expressions to locations. Locations may beaggregated into regions, and a concept-location relationship may furtherbe associated with a time indication. That is, concepts associated witha location may change over time such as throughout the day or week ormonth.

The method and systems described herein provide creating an accuratethought map by recording for analysis people's many utterances ofnatural language expressions (hereafter “natural language expressions”or “expressions”), each expression recorded together with the locationat which each expression was made. Natural Language Understanding (NLU)techniques are used to interpret a person's text or spoken words,creating an interpretation data structure to represent the automaticallydetermined semantics of an expression. A person's context and/orrelationships among words in a sequence of words may be used to enhanceunderstanding of the expression. Entire interpretation data structuresof a natural language expression may be stored in a database togetherwith the associated location data. Statistical techniques or other dataaggregation methods may be used to analyze these database records todetermine concepts frequently associated with a place of interest. Onceone or more concepts are associated with a location (and possibly atime), the data may be used to discover which concepts are associatedwith a particular place. A person considering buying a house at aparticular address may get an idea of the neighborhood dynamics bylooking at the prevalent concepts in the area. Alternatively, todiscover which places are associated with a particular concept, whichmight be useful for determining where to place certain advertisements.

Terminology

Thoughts: non-technical term referring to what's on people's minds whenthey speak or write natural language. Thoughts are represented usingcomputer data structures that represent the semantic interpretations ofnatural language expressions.

Region: a geographic range that can be displayed on a map. Regions maybe defined in many ways. For example, one or more of latitude-longituderanges, zip codes, geopolitical definitions or custom boundaries, suchthe locations within view, or within hearing distance, of a specificpoint of interest.

Location: a smallest distinguishable area within a region, such as acell within map grid, a building, or an area within hearing range of amicrophone.

Geolocation: a specific point within a region, such as a specificlatitude-longitude pair, specific zip code, or specific point ofinterest. Geolocation data is fresh if it has been updated more recentlythan the amount of time it could take to change location. Thegeolocation for a stationary device is always fresh. The geolocation fora slow moving device, such as a hiker's mobile phone in a large mountainrange is fresh for long periods of time. The geolocation for a fastmoving device, such an airplane over a city, is fresh only for shortperiods of time.

Interpretation: an interpretation is a complex data structure created bya semantic parser and used by a data processing system to representmeaning of natural language expressions.

Component: A component is a part of an interpretation that represents aunit of meaning. In semantic graph, a component is a subgraph thatcomprises a subset of nodes and edges in the interpretation; thissubgraph represents entities and relationships between entities thattogether define the component. The smallest components involve a singlenode (an entity), or two nodes joined by an edge. For example, acomponent may represent the relationship between a person entity who isthe parent and a person entity who is the child.

Constituent: Constituents are words or group of words that function assingle units within hierarchical structures (such as clauses andphrases). Natural language grammars, including primarily phrasestructure grammars (e.g., context-free grammars) and dependencygrammars, break down expressions into constituent parts. The structurescreated by these two types of grammars are different, although closelyrelated.

All constituents are explicit parts of the original expression; thecorresponding components are semantic counterparts of thoseconstituents.

Domain of discourse (knowledge domain): subject area that is associatedwith a class hierarchy of graph. For example, a food domain may includea fast food class that may include a hamburger class and a french friesclass.

Concept: an abstract idea, a general notion associated with an entity orclass. Fast food, hamburgers and french fries may be concepts in a foodknowledge domain.

Expression: People can use natural speech to create spoken expressions,and write, type, or gesture natural language expressions. Microphones,keyboards, touch screens, cameras, and other means are appropriate forcapturing expressions. Expressions can be created in text form.Expressions can be converted to text form by automatic speechrecognition (ASR) systems.

Salience: the difference between a localized density and a generaldensity of a concept being expressed.

Person: a source of natural language expressions.

User: somebody who operates a machine, system, or service such as amobile phone, the internet, or a virtual assistant.

Consumer: the target of advertising. Some embodiments processexpressions from some people to determine advertisements to present toconsumers. In some cases the people who make the natural languageexpressions are the consumers.

Ontology: An ontology represents at least a collection of object classesand relationships. between instances of the classes. In someembodiments, ontology includes a hierarchy of classes. In someembodiments, a relationship is constrained by a type system thatspecifies the classes of the instances that can enter the relationship.Some embodiments associate classes with behaviors called ‘scripts’ thatdescribe sequences of actions that an instance of the class can perform.

The present disclosure relates to various aspects of data structures,data collection, and data analysis. These are applicable to variousindustries and for various purposes, such as providing people withuseful information just when they need it. Not all aspects are presentin all embodiments of the invention described and claimed.

An aspect of the invention is capturing and processing natural languageexpressions. An aspect is collecting data using audio capture, such asthrough microphones. An aspect is capturing audio using mobile devices.An aspect is capturing audio using geostationary devices. An aspect iscapturing audio pervasively, in places where people are and travel.

An aspect is associating natural language expressions with particularlocations. An aspect is associating natural language expressions withparticular time ranges. An aspect is associating natural languageexpressions with particular people. An aspect is associating naturallanguage expressions with particular demographic information. An aspectis associating natural language expressions with particular devices. Anaspect is associating natural language expressions with measures ofaccuracy of geolocation. An aspect is associating natural languageexpressions with measures of the age of geolocation detection. An aspectis associating natural language expressions with measures of speed anddirection of movement.

An aspect is using a computer for the natural language understanding ofexpressions to determine interpretations. An aspect is processingnatural language expressions to determine emotions of speakers. Anaspect is matching parts of interpretations. An aspect is ignoringparticular entities and attributes within interpretations.

An aspect is storing geolocation information in a representation that isconducive to rendering maps at different scales. An aspect is filteringthoughts based on values or ranges of location, timestamp, cyclicalphase, person, device, and other parameters associated with naturallanguage expressions. An aspect is determining locations of thoughtsources. An aspect is determining location by using averaging ofmultiple measurements. An aspect is weighting averaged locations byaccuracy of location measurement. An aspect is correcting locationmeasurements by adding direction vector times speed times measurementage. Another aspect is performing fuzzy matching of location.

An aspect is predictively providing useful information. An aspect isproviding location-specific advertising. An aspect is providingperson-specific advertising. An aspect is providing time-specificadvertising. An aspect is providing information that promotes publicsafety. An aspect is predicting future behavior of people. An aspect islooking up particular thoughts or categories of thoughts according togeolocation, time of day, day of week, or other criteria.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system in which a network connects people, dataminers, and analysts to servers according to an embodiment of theinvention.

FIG. 2 is a flow diagram that illustrates the process of capturing andanalyzing data to create a thought map according to an embodiment.

FIG. 3 illustrates an embodiment of a person-facing subsystem, andcommunication with a data collection subsystem according to anembodiment.

FIG. 4 illustrates a parse tree for an expression according to anembodiment.

FIG. 5 illustrates a dependency tree for the same expression accordingto an embodiment.

FIG. 6 illustrates an interpretation data structure for the sameexpression according to an embodiment.

FIG. 7 illustrates an integrated embodiment of a data mining andanalysis subsystem according to an embodiment.

FIG. 8 illustrates map layers, including a geomap, an emotion overlay,an overlay of filtered expression points, and an overlay of conceptwords according to an embodiment.

FIG. 9 illustrates a geomap with indications of locations of naturallanguage expressions according to an embodiment.

FIG. 10 illustrates a geomap with indications of concepts and locationswhere they were expressed according to an embodiment.

FIG. 11 illustrates locations of expressed concepts according to anembodiment.

FIG. 12 illustrates locations of expressed concepts filtered for amorning time range according to an embodiment.

FIG. 13 illustrates locations of expressed concepts filtered for anafternoon time range according to an embodiment.

FIG. 14 illustrates locations of expressed concepts filtered for anevening time range according to an embodiment.

FIG. 15 illustrates a geomap within a billboard advertising beer, andexpressed concepts according to an embodiment.

FIG. 16 is a block diagram of a computer system, according to anembodiment.

DETAILED DESCRIPTION

The present invention relates to creating thought maps for a geographicregion, optionally over time. The value of the thought maps depends onthe accuracy of mining concepts from natural language expressions ofpeople in the locations being mapped.

FIG. 1 illustrates a system in which a network connects people, dataminers, and analysts to servers, according to an embodiment of theinvention. Natural language expressions are collected from mobile deviceusers 701 and 702, and users of devices with known fixed locations 703.Known locations are often apparent from an IP address, softwareregistration, cell tower proximity, or other known methods. Auser-facing server subsystem 704 receives the user expressions from overa network, and passes the expressions to servers in a data collectionsubsystem 705. In some embodiments, the user-facing subsystem 704 parsesand interprets expressions to produce interpretations, and in someembodiments the data collection subsystem 705 parses and interpretsexpressions to produce interpretations. The data collection subsystem705 add records to a database 706 comprising raw data from expressions,their context, and their interpretations. Servers within a data miningsubsystem 707, as directed by data miners 708 and 709, process the rawdata to create a database of mined data 710. Analysts 711 and 712 accessthe mined data using servers within an analyst-facing subsystem 713.Various embodiments have any number of users, data miners, and analysts.

The data mining subsystems may proactively transform the raw data indatabase 706 into a format more easily used for analysis purposes andstored in mined data 710. Alternatively, the analyst-facing subsystemsmay start with the raw data in database 706 instead of using pre-mineddata.

In one embodiment, an organization providing the data collectionsubsystem 705 may be distinct from the organization providing the datamining subsystem 707. Yet a third distinct organization may provide theanalyst-facing subsystem 713. In other embodiments, a singleorganization may provide two or all of these subsystems. Similarly, somepeople may perform both data miner and analyst roles, and some peoplemay perform one role and not the other. Many variations of theembodiment of FIG. 1 will be apparent to ordinarily skilledpractitioners.

FIG. 2 is a flow diagram that illustrates the process of creating athought map according to an embodiment of the invention. People locatedin a region of interest speak in the presence of a microphone or type anatural language phrase on a device such as a phone. These naturallanguage expressions may be processed by a local computing device, forexample, a device that captured the speech or text. Alternatively, theexpressions may be sent to a data center outside of the region forprocessing. The NLU system 95 receives the expressions along with anindicator of the location in which the expression was captured andcreates an interpretation, which is a semantic representation for eachexpression. The NLU system 95 stores the interpretation data structurein a database, such as Raw database of interpretations in content 93,together with the associated location.

A data mining subsystem 91 analyzes the data in raw database 93 todetermine for each distinct location represented, concepts in eachexpression, and aggregates the concepts across users. Data miningsubsystem 91 searches the interpretation data structures for concepts ofinterest. The concepts of interest are stored in one or more knowledgedomain concept graphs.

Data mining can be run proactively at a configured interval, or can betriggered on demand by a data-mining request that may specify one ormore knowledge domains. An example aggregation is to count the number ofoccurrences of a concept across all interpretations associated with aparticular location, or to count the number of interpretations thatinclude at least one instance of the concept.

The output from the data mining process is stored in the database ofMined data 94. This data provides sufficient information to draw athought map of the region of interest. However, an analyst-facingsubsystem 92 may retrieve data in response to a query created to answeran analyst request. For example, an analyst might request a histogram ofthe frequencies of the top 100 most frequently expressed concepts. Inanother example, an analyst might request a list of the 10 most commonconcepts expressed by people in the age range of 21 to 35 years old. Inanother example, an analyst might request the salience of a specificconcept within a location.

The present invention, in various embodiments, relates to the processingof natural language expressions to serve the needs of multipleconstituencies, such as users of mobile devices, data miners, analystsand advertisers.

FIG. 3 illustrates an embodiment of a mobile-device-user-facingsubsystem and its communication with a data collection subsystem.Natural language expressions may be spoken and captured using amicrophone in a mobile device such as a cell phone, or from stationarymicrophones installed in locations of interest. Any means of capturingspeech audio signals can be used. The acoustic signals may betransmitted over a network the user-facing subsystem 704. Alternatively,when the microphone capturing the speech is connected a processor, theprocessor may transcribe the captured speech locally on the device, andthe resulting textual transcription may be sent over the network to theuser-facing subsystem 704. Natural language expressions may be typed astext, requiring no speech to text transcription.

Within user-facing subsystem 704, a speech recognizer 81 converts speechto text. A semantic parser 82 analyzes text from people's expressions tocreate corresponding interpretations. The text may come from the outputof the speech recognizer 81, from a speech transcriber co-located withthe speaker (such as a mobile device), or from the person expressingthoughts directly in text. The semantic parser uses contextualinformation, including one or more of geolocation, time, and useridentity, among other information, to improve its parsing ability. Moredetailed explanation is provided about contextual information below.

In some embodiments, and for some expressions, an expression executionmodule 83 uses the interpretation to construct a response for the user.

In response to receiving an expression, the user-facing subsystemassembles information packages that include one or more of theexpression audio, expression text, an interpretation of each expression,concepts from the interpretations, and expression context information.The subsystem sends the information package to a data collection system84. Various embodiments include different content in informationpackages. In some embodiments, information packages have a richstructure. Some embodiments serialize information packages whentransmitting them between subsystems. In some embodiments, serializedinformation packages are represented using special purpose formats. Somesuch formats use custom implementations. Some use schemas such as JSON.In some embodiments, serialization and de-serialization are notnecessary because the user-facing systems and data collection systemsare hosted by the same servers and can share internal data structures.

The data collection system 84 receives the information packets, andstores their contents in a database 85. Many alternative formats arepossible for storing records in the raw database. Some of the data to bestored, such as time, location or user ID, has a fixed size. Otherpieces of information, such as expression audio, expression text or someof the constituents, can be stored as variable strings. (“[[Ed] wants[to impress [[the] girl [with [[a] [pearl] necklace]]]]]”) is a simplestring format for representing a dependency tree. In some embodiments,such strings can be annotated to associate tree nodes with semanticdata, such as constituents, or references to constituents through uniqueidentifiers. In some embodiments, interpretations are stored using setsof triples. Some embodiments index the database proactively when addingrecords to the raw database. Other embodiments create indexes at a latertime.

Contextual Information

A variety of contextual information may be captured in association witha natural language expression including location, personalization, andtime. The contextual information may be used to increase the accuracy ofconstructing an interpretation or mining concepts from aninterpretation. Interpretation records may also be stored as indexed bycontextual information to enable fast retrieval of interpretationrecords relevant to creating a particular kind of thought map.

Many notations exist to represent geolocation includinglatitude+longitude, Geohash, what3words, street addresses, and zipcodes. Some embodiments store geolocation data along withinterpretations in database records. Some embodiments index the databaseby geolocation. The Geohash encoding of locations is of particularinterest due to the combination of its ease of searching and itsscalability. A Geohash represents a two-dimensional rectangular range oflatitude-longitude values as a string of lower case characters; thelength of the code determines the resolution of the cell. Geohash codingsubdivides space into a hierarchical grid of cells. Each additionalcharacter in a Geohash code divides the current cell into 32 sub-cells,alternately 4×8 or 8×4 sub-cells, so that at each stage, “portrait”cells alternate with “landscape” cells. Conversely, removing charactersfrom the end of a Geohash code loses precision gradually. Anotherencoding known as quad trees has similar hierarchical properties, butthe encoding uses binary strings.

Many systems can contribute to the determination of geolocation; theseinclude satellite constellation systems such as Global PositioningSystem (GPS), Galileo, Glonass, and BeiDou, Long Range Navigation(LORAN), cell tower proximity and triangulation, and Bluetooth LowEnergy. Some systems can also determine a degree of accuracy for eachlocation measurement. Some systems only measure location from time totime, and report a measurement age when the location value is retrieved.Some systems also measure speed of motion. Together, speed and age areuseful to estimate geolocation at a time different from the moment ofmeasurement.

Some embodiments store one or more of geolocation, location measurementaccuracy, measurement age, and speed with each record in a database.Some embodiments measure or provide geolocation information with aprecision that is less than the measurement accuracy. This is useful forobscuring positions of measurements taken. Some such embodiments roundor truncate coordinates, which causes banding in representations. Someembodiments add random values within a small range to measurements inorder to reduce accuracy.

A user device that captures a user's expressions may also providepersonalization data that identifies the person who articulated theexpression. This is only possible if systems can identify peopleuniquely, such as personal mobile devices or systems that require userauthentication such as ATMs. If the system captures expressions in apublic place, such as a shopping kiosk or vending vehicle, or on adevice shared by many people such as an office equipment system, uniqueuser IDs are unavailable.

Some embodiments, whether or not they identify the speaker who producesan expression, identify listeners. Some such embodiments store, withrecords in the database, a list of person IDs of listeners known to bepresent, if any. Some embodiments that identify listeners maintain aseparate “listener” database, and include date-ordered lists of recordsthat were likely heard by the person. This is useful to assist humanperception and memory by capturing expressions, ensuring that thelistener did not mis-hear them, and recording them in personal knowledgebases for future search and recall. Some such systems interface topersonal cognitive enhancement devices.

Some embodiments store, with each record in the database, demographicinformation about the speaker. Demographic information includesinformation such as gender, age, marital status, race, politicalorientation, education level, and economic status. Some such embodimentsindex the interpretation database by one or more types of demographicinformation. This enables better prediction of people's interests,needs, and likely activities, which enables various devices to providebetter services. It also enables targeting of advertisements and lawenforcement.

In an embodiment, each expression record in the database may be storedwith a timestamp identifying the time at which the expression wasuttered. In such an embodiment, the database records may be indexed bytimestamp. This is useful for efficiently filtering records by timerange.

Some embodiments store, with each record in the database, time periodphase information. This is useful for filtering records based on timeranges within event cycles, such as ranges of hours within days, dayswithin weeks, days within years, or months within years. Someembodiments calculate phases from timestamps after reading the database.Some embodiments record phases in the database.

Creating Interpretations

This section describes the NLU System 95 that receives natural languageexpressions and creates interpretations to store in raw database 93.

People communicate thoughts using natural language expressions, usingspeech or text. Computer systems can represent human thoughts using datastructures that provide computable representations of meaning. Suchsemantic representation data structures are called interpretations ofthe expressions. Interpretations may be stored in various as datastructure formats. Common interpretation formats include frames (asdescribed by Marvin Minsky's 1974 article “A Framework for RepresentingKnowledge”), semantic memory triples <Entity, Attribute, Value> (EAVtriples), and first-order logic. The various formats used forrepresenting meaning are broadly equivalent and largely mutuallyconvertible, although they can have different advantages and differentexpressive power. In this disclosure, example interpretations arerepresented using triples in a semantic memory. Other semanticrepresentations are appropriate for practicing the invention.

A triple consists of an entity, a named attribute, and one or morevalues of the attribute. An entity is an instance of a class in theontology of a domain, representing an object of discourse—for example,an animate object, an inanimate object, or an abstract object. Entitiessuch as people, things, places, and times, appear in natural languageexpressions, and may be specified by nouns, such as “Ulaanbaatar”, or byphrases, such as “the capital of Mongolia”.

Defining a class hierarchy is an essential part of an ontology. Forexample, in a virtual assistant's Calendar domain, a Woman is a Person,and a Person has attributes such as Name, PhoneNumber, Address, Gender,and so on. Whereas in a general Biology domain, a Person is a Human, aHuman is a Primate, a Primate is a Mammal, and so on. This is asimplistic taxonomy. A Mammal has a Gender attribute, for example, andall the subclasses of Mammal have a Gender attribute that is inheritedfrom the Mammal class. In some embodiments, an ontology specifies atleast a class hierarchy, a set of attributes applicable to each class,and constraints (such as types) on the values of attributes.

The interpretation of a given expression often comprises severalentities, and each entity may have multiple known attributes. Forexample, an act of eating may involve the person who does the eating anda description of the food that is eaten. The person is an entity thathas a name, which in various embodiments may be represented by one ormore of a string, a complex data structure, and a user identifier (UID).A geolocation and time of day may also be associated with theinterpretation of the expression as a whole.

An entity of type Thing (i.e., a physical object) can have numerousattributes; only some of them are applicable to each subclass of Thing.Possible attributes of a Thing include a size, a material, and a list ofparts. The parts are instances of other Thing entities. In someembodiments, the interpretation data structure simply omits attributeswhose values are unknown. Place entities can have attributes such as anaddress, latitude, longitude, and a name. Time entities can have year,month, day, hour, minute, second, and a time zone.

Each instance of an entity has one or more attributes, each of which hasa value that, at any given time, may be assigned or not. Most valueshave types; some types have range restrictions. For example, an agevalue may be a number of years, from 0 to 122. A gender value is aselection from among a list of discrete choices, and a name value is oneor several words.

For another example, consider that a Person entity might have a “shirt”attribute that references a Shirt entity, which (as any subclass ofClothing) supports a “place of manufacturing” attribute, whose value isa Country entity, and a Country has a “name” attribute. Thus, after thePerson referred to by “her” has been identified, a system can answer thequestion, “Where was her shirt made?”

Regardless of the particular knowledge representation in an embodiment,interpretations provide a way to expose the structure of expressions.Some embodiments structure interpretations as parse trees, syntax trees,or dependency trees. Some embodiments augment nodes and edges withsemantic information. Some embodiments represent the semantic structureusing property lists.

In some embodiments, the interpretation of a natural language expressionis created by a semantic parser. In some embodiments, the semanticparser extracts the expression's syntactic structure, such as adependency tree, according to a grammar, such as a dependency grammar.Given the expression, the grammar creates a hierarchy of constituents,and the semantic parser associates with each constituent a correspondingsemantic component. A constituent is a part of the natural languageexpression, and a component is a part of the semantic interpretation ofthe expression. In some embodiments, the semantic parser depends onpartial parsing techniques. Some such embodiments use part-of-speech(POS) tagging to identify some, but not all, of the constituents andassociated semantics. Some embodiments do partial parsing. Someembodiments do full parsing.

FIG. 4 illustrates a possible parse tree for an expression “Ed wants toimpress the girl with a pearl necklace” according to a phrase structuregrammar. FIG. 5 illustrates a possible dependency tree according to adependency grammar for the same expression. A “constituent” is eitherthe entire expression, or an expression generated by any labeled subtreethat consists of a node plus all the nodes dominated by that node.Comparing FIG. 4 and FIG. 5 shows that most, but not all, constituentsare the same in the two tree structures. For example, “the girl with apearl necklace” is a constituent in both trees, but “girl” is aconstituent in the phrase structure grammar only. The essentialconstituents from the point of view of building a semanticinterpretation are the dependency constituents. In the presentdisclosure, the hierarchy of dependency grammar constituents is conveyedtextually using brackets that mimic the dependency tree:

[[Ed] wants [to impress [[the] girl [with [[a] [pearl] necklace]]]]].

The corresponding dependency grammar constituents are enumerated below:

“Ed”

“the”

“a”

“pearl”

“a pearl necklace”

“with a pearl necklace”

“the girl with a pearl necklace”

“to impress the girl with a pearl necklace”

“Ed wants to impress the girl with a pearl necklace”

The role of constituents and their associated components can beillustrated by comparing interpretations of an ambiguous expression. Anambiguity is present in the example “Ed wants to impress the girl with apearl necklace.” The pearl necklace might be something distinctive thatthe girl wears that identifies which girl Ed wants to impress.Alternatively, Ed might want to impress the girl by giving her a pearlnecklace. The ambiguity leads to two ways of breaking the sentence intoconstituents. The contrast between the two constituent structures isshown below, by displaying the groupings where the difference occurs:

(1) Ed wants [to impress [the girl] [with a pearl necklace]]

(2) Ed wants [to impress [the girl with a pearl necklace]]

Structure (2) is the one that had been illustrated in FIG. 4 and FIG. 5.The two different structures convey different meanings. For example,using structure (1), the question “How does Ed want to impress thegirl?” can be answered “with a pearl necklace” whereas using structure(2), the same question could be answered “I don't know”. The question“Who did Ed want to impress?” is simply answered “the girl” usingstructure (1), and “the girl with a pearl necklace” using structure (2);in a given context, these might be different girls, or perhaps “girlwith a pearl necklace” could be interpreted as referring to a specificgirl, whereas “the girl” could remain ambiguous. Using structure (1),the question “Is Ed interested in buying a pearl necklace?” is answered“Yes” (meaning: this is likely under the circumstances) whereas usingstructure (2), the same question is answered “No” (meaning: there is noreason to think so), and the difference between the answers is relevantto an advertiser of pearl necklaces.

FIG. 6 illustrates a possible interpretation structure, in anembodiment, corresponding to FIG. 4 and FIG. 5, and to structure (2)above. Unique entity IDs such as V00315 (#31), and P00175 (#32), arecreated by the system as needed to represent objects of discourse. Theyappear as nodes in the representation. In this semantic data structure,entity V00315 [Reference number (#) 31], is an instance of the Verbclass. As such, it has an associated verb, and verb properties. The verbis “to want” and the properties (#33), are: present tense, third person,singular.

The verb “to want” associated with Verb instance V00315 (#31), has asubject (the one who wants) which is associated with the constituent“Ed” (#34) and has been paraphrased in the graph as “a Person calledEd.” Verb instance V00315 (#31), also has an object, which specifieswhat is wanted, specifically for this instance. The wanted entity, inthis case, is Verb instance V00206 (#35), which is the verb “to impress”in the infinitive form. The verb “to impress” inherits its subject fromthe governing auxiliary verb “to want” and has an object, the Person tobe impressed, which is represented by the substructure whose entityunique ID is P00394 (#36). This entity does not have a name, but itstands for the constituent “a girl with a pearl necklace” and it isparaphrased in the graph as “a female Person who wears thing T00123(#37).” The graph specifies that T00123 (#37) is an instance of theclass PearlNecklace, and it is further known to the system (by way of adomain ontology) that PearlNecklace is a subclass of the PearlNecklaceclass. The semantic network shown in FIG. 6 illustrates the keyrelationships that are relevant to applications involving this example.Other embodiments represent relationships in different ways.

In some embodiments, co-reference resolution techniques are used in anattempt to equate P00175 (#32) (“Ed”) or P00394 (#36) (“the girl withthe pearl necklace”) with previously known entities. Co-referenceresolution is performed based on available context. Context can includeprevious expressions, knowledge about the environment, or other relevantfacts. Any co-reference resolution technique is appropriate.

All constituents are explicit parts of the original expression; thecorresponding components are semantic counterparts of thoseconstituents. In some embodiments, components are complex substructures,such as subgraphs, with a head node; for example, the head node for “toimpress the girl with the pearl necklace” is P00394 (#36).

A concept is an abstract idea, a general notion associated with anentity or class (such as a fruit), an action (picking a fruit) or anentire scenario or “script” (a formal wedding). A concept belongs in aknowledge domain. Some concepts are expressed directly by words orphrases in an expression. For example, a mention of “apple” expressesthe concepts “Apple”. But it also invokes the concept Fruit, which is asuperclass of “Apple” in a domain ontology. The use of domain ontologyincludes ascending ontology classes from more specific to more genericclasses. Note that ontology is not a strict hierarchy. The Person classis a subclass of Human, itself a subclass of Mammal, Animal, and so on,but it is also a subclass of Contact, a class that also has Business asa subclass. More advanced uses of ontology depend on associating ascript with the meaning of an expression. For example, the expression“The masked man broke into WellsFargo through the roof” calls for theunderlying script Robbing_A_Bank_, which is a subscript of theRobbery_ontology script. The verbalization of a concept uses conceptwords, in this case, “robbery” and “bank” or “bank robbery”; these wordsare only partial reflections of the concept such as theRobbing_A_Bank_script.

To simplify communication, it is common to refer to the concept words (aconcept's verbalization) as if they were the concept itself; theconcept, internally, can be a complex data structure that presents achallenge to communicate and display.

Another example, in another domain of discourse, is the concept“measuring the blood pressure of a patient.” Other concepts include“measuring blood pressure” and “pressure measurement” however “measuringblood” is not a concept; this is because it is not blood that ismeasured, only its pressure. Measuring blood is not semanticallyself-contained, generic and broadly useful. The essential properties ofa concept are that it can form a part of a semantic interpretation, andthat it can be verbalized.

Specific representations of interpretations given above are merelyillustrative of a full syntactic and semantic treatment of naturallanguage expressions.

Representations of parse trees (FIG. 4), dependency tress (FIG. 5) andinterpretation graphs (FIG. 6) typically use internal pointers forefficiency, but they can be serialized in order to be transmitted to aseparate part of a larger system, and de-serialized by the receivingsystem module. Languages often used for serialization/deserializationinclude JSON and XML.

Emotional Charge and Emotions

Some embodiments analyze natural language expressions for emotionalcharge, which they store either within or in association with theinterpretation. There are various methods of sentiment analysis. Someembodiments assign an emotional charge to a concept in the form of asingle number. Such a representation of emotional charge can assume arange of values from highly negative to highly positive. For example,“killing” and related concepts have a highly negative charge, whereas“love” has a highly positive charge. Emotional charge can also be scaledaccording to factors that depend on values of attributes. For example,if the value of the score attribute of a school exam is ‘so-so, theconcept is a little bit negative. If the score is ‘good’, the concept isa little bit positive. If the score is ‘excellent’, the concept is verypositive. Some embodiments compute the emotional charge of expressions.Some embodiments scale the emotional charge using contextual informationabout people and their environments. For example, “money” has differentemotional charges to different people, and “deepness” has a differentemotional charge in swimming than in shoveling snow. Some embodimentstrain the association of emotional charge with concepts or scalingfunctions by means of crowdsourcing and machine learning. Aftertraining, the value of emotional charge can be derived, at least inpart, from the trained associations.

Some sentiment analysis functions represent simply a scale of 1 star to5 stars. Some sentiment analysis functions represent emotions aspolarity and magnitude values. Various systems exist for representingemotion values. Some quantify values of emotions such as anger, disgust,fear, happiness, sadness, and surprise; others are dimensional, usingtwo, three, or more dimensions. Various embodiments processinterpretations or associated concept sets to determine emotion values.Some embodiments use attributes of captured speech, such as prosody,including speech rate and rhythm, stress, emphasis and pitch contour, tocompute or scale emotion values. Some embodiments store emotion valuesalong with interpretation in database records.

Advantages of Concepts Over Keywords

Extracting keywords from expressions is trivial. Keyword descriptionsare not sensitive to word order. For example, the expressions “Ask whatyour country can do for you” and “Ask what you can do for your country”have the same keywords, though they have different meanings. Similarly,“a little too big” has both keywords ‘little’ and ‘big’, but the meaningrelates to being big, not little. In Chinese, “

” and “

” have the same set of characters, though they have different meanings.In French, “l′amour fait passer le temps” and “le temps fait passerl′amour” have the same keywords, though they have different meanings.For many expressions, parsing the natural language is required todetermine the intended meaning.

In contrast with simply extracting keywords, a semantic parser thatconverts natural language expressions to semantic interpretations iscomplex, but represents thoughts more accurately and is better able tounderstand the intent of an expression. A correct interpretation of “Theguy who says D'oh” is the cartoon character Homer Simpson, though“Homer” is not a keyword in the expression. An appropriate system forinterpretation determines so through parsing, semantics, and triviaknowledge.

Consider a location where people negotiate or work on projects in turns,and they frequently say, “the ball's in your court”. Keyword-basedmapping will give false positive results to sports-related concepts.

Consider a location where certain people talk about clothing and oftensay, “I like that on you.” Keyword-based mapping will fail to identifythe concept of clothing.

A benefit of using natural language interpretation over keywords inemotion analysis is that it is possible to determine sentiments forparticular entities, rather than simply in particular locations.Determining, for example, that people in a kindergarten love Elmo dolls,people in a high school laugh at Elmo backpacks, and people in an officebuilding are annoyed by Elmo pop-up ads in web browsers can helpadvertisers decide where to place billboards with Elmo, how prominentlyto feature Elmo on the billboard, and how much to bid for the trademarkrights to use the image of Elmo for advertising products.

Systems according to the invention, which use interpretations, canidentify concepts implied by non-key words, and disregard falseconcepts, such as ones resulting from idioms. Therefore, mapping byinterpretations produces more accurate thought maps than conventionalkeyword methods.

Database Organization

Some embodiments, such as those that require real-time analysis, provideinformation, identify emergencies, or take actions responsive todynamically changing situations. Doing so requires fast data access andanalysis. Such embodiments may use indexed real-time databases, such asStructured Query Language (SQL) databases localized to a single server.Using a single server avoids incurring latency for communication betweenservers.

Some embodiments are capable of storing more records than can be storedon a single server. In various embodiments, interpretation records aredifferent sizes, and typically in the range of several kilobytes perrecord. Some embodiments are able to store trillions of interpretations.In some embodiments, speech recognition, interpretation, or otherprocessing before storage requires several seconds of processor time.Such systems require widely distributed storage and processing. Aframework of unindexed distributed database records such as Hadoop orother MapReduce type of system may be appropriate. Reading, filtering,and processing records from such distributed systems might take hours insome cases.

Some embodiments support both real-time and massively distributed datastorage and access. Some such systems store recently captured records inan indexed real-time cache database and write the records to adistributed database after some time. Some such systems, when aninterpretation is retrieved from distributed data storage, other recordswith the same or similar interpretation are loaded from the distributeddatabase. Some such systems, in response to capturing expressions in onegeolocation, load historical expressions within a particular distance ofthe geolocation. For faster determination of records to load into thecache, some systems restrict requests to geographically local datacenters, data centers associated with a particular user ID, or datacenters selected according to other environmental information. Some suchsystems maintain a score for each record in the cache database. Thescore depends on the age of the record access, the proximity of therecord capture location to the location of one or more people, the phaseof capture time of the record, and various other relevant information.When a new record arrives from the distributed database, the cachedatabase evicts the record with the lowest score.

Some embodiments store interpretations based on location. This can havebenefits in access speed. A geographic region is divided into stripsalong one range of longitude, with the strip width chosen to allow asmall enough datasets for fast access to a selected strip.Interpretations may be sorted based on their position along their strip.Each strip is stored separately, such as on separate storage media. Asearch of a region spanning multiple strips can read the appropriaterange of interpretation data within a strip simultaneously from each ofthe strip storage arrays. A search for a region smaller than a strip canget all data for a search from a continuous range within a single strip,or from two strips, if the location range happens to cross the boundarybetween strips. In some embodiments, strips consist of ranges oflongitude. In some embodiments, the ranges are non-identical and chosenin order to ensure an approximately consistent number of interpretationsin each range. This generally corresponds to the population of capturedspeakers within each longitudinal strip. Some such embodimentsoccasionally rebalance storage allocation by moving interpretation datafrom one strip to another. A strip to which new interpretations areadded may be re-sorted. Some such embodiments make it possible toprovide real-time advertising and virtual assistants for billions ofpeople based on trillions of stored expression interpretations withpractical numbers of storage devices and internet bandwidth.

Some embodiments represent locations on a grid of cells and provideseparate data stores for each grid cell. Therefore, for search rangesless than a single grid cell, a search function might need to read fromone, two, or four grid cell stores, based on whether the range crossesnone, one horizontal or vertical, or both of a horizontal and verticalgrid cell boundary. Some embodiments use grid cells of varying sizes,using larger cells for areas of lower expression density in order tomake storage groups have approximately consistent amounts of data. Someembodiments compose large grid cells as collections of abutted smallgrid cells.

Some embodiments using either strips or grid-based storage, may segmentdata by region, such as for individual cities. The storage devicesstoring data for a particular location may be placed in data centersclose to the locations represented by the data in order to minimizeaccess latency and long-distance network bandwidth.

Some embodiments segment data in radial segments from a central point.Strips are thereby rings of specific widths. Like longitudinal strips,ring widths do not need to be consistent, though regular widths simplifyand therefore speed up calculations of which ring in which to store andretrieve interpretation data. Some embodiments store rings in which thewidth of a ring is proportional to distance from a center. A benefit ofrings is that, by placing the center at the center of a dense area,rings of consistent width will naturally tend to have similar amounts ofdata. Some embodiments use rings in and around urban areas, but forlocations that fall outside of any urban ring, data is sorted bylongitudinal strips. This keeps computation fast for most accesses, butrequires only the more complex calculation in the exceptional cases ofuses distant from urban areas.

Data Mining

Processing the raw database records may occur at different timesincluding when a new expression is captured, when an analyst creates athought map; and at intermediate times. Intermediate processing stepsconsume data from raw databases and transform it into data in a datamining database. This section describes the functioning of the datamining subsystem 91. The data mining subsystem retrieves records fromthe raw database, performs indexing, aggregation, and correlation on thedata, and stores the resulting transformed data in a data miningdatabase for use by analysts.

Not all records stored in the raw database are used to create mineddata. Some embodiments ignore database records when their geolocationuncertainty exceeds a certain fraction of the display size becausegeolocation records of such interpretations are not meaningful. Someembodiments ignore or discard records whose age measurement exceeds acertain age limit, because old measurements are also inaccurate. Due tothe world being ever-changing, the information value of expressioninterpretations decreases over time and becomes stale. Some embodimentscombine age with a speed measurement: the faster the motion, the morerecent measurements must be in order to be considered useful. Certaintypes of values fade out according to a known amortization scheme, suchas decay, where the rate of decay depends on the type of information.Other embodiments remove values according a fixed expiration date, theduration of validity depending on the type of information.

Some embodiments perform data mining autonomously. Much of such datamining activity relies on scripts or programs that proactively andunconditionally perform certain analyses of the raw data. Someembodiments run analyses either periodically or in response to events.Some such embodiments perform analyses incrementally as they acquire rawdata.

Examples of autonomous processing of raw database records includeindexing a raw database according to values such as location, time,presence of syntactic constructs or semantic concepts, and any otheraspects of interpretations. They also include collecting statistics ofdata, such as occurrence counts and histograms, concepts, location rangedata, time range data, multivariate histograms that cross-analyze two ormore of the dimensions, multivariate histogram density estimation, andvarious other joint statistics of multiple measurements, including meansand variances of the above data.

In some embodiments, data mining specialists drive data miningoperations such as those described above and others that are morespecialized. The mined data may be derived from the raw data by eitherautomated scripts or commands from data miners. The embodiment collectsthe mined data 94 in a database of results. Some embodiments storegeographical information in a spatial database such as one using thePostGIS format, in which functions such as distance, area, union,intersection, and specialty geometry data types exist, and can be addedto the database. A spatial database is useful to store and retrieve datathat represents objects defined in a geometric space. PostGIS is an opensource and freely available spatial database extender for the PostgreSQLDatabase Management System.

Some embodiments compute salience for specific concepts as an indicationof their local relevance. Salience indicates a concept's relevance in aplace of interest as relative to other places. Salience is useful as avalue on which to filter map results. It allows analysts and data minersto focus their work on the most valuable concepts and locations.

Some embodiments derive salience from 2-D density data as follows. Asystem obtains a measure of salience for a concept (such as a histogramcount, weighted by a relevance weight) as the difference between twodensities. The density of the concept at a small scale (averaged over asmall range) is known as a narrow density. The density of the concept ata large scale (averaged over a large range) is known as a wide density.In some embodiments, the salience of a concept is the difference betweenthe narrow density and the broad density. This corresponds to the use ofa “center-surround” filter. Many variations of this design are possible.Some embodiments use the ratio of the two densities, rather than theirdifference. One reason is that, for making decisions against athreshold, the absolute density of a concept is not as important as itsdensity relative to the background. Using ratios is closely linked tousing the logarithm of densities. This presents some issues fornear-zero densities. When using a ratio, it is advisable to add a biasterm to both the numerator (narrow density) and the denominator (widedensity) before the division, to avoid issues with small denominators,which cause ratios to shoot up. Using a large bias has the effect thatthe results of the ratio calculation approach those of the differencecalculation. An intermediate value of the bias is best, such as a valuescaled by the global statistics of the data, e.g. a median value ofweighted saliences.

Salience, as described above, is a function of two scales, a narrowscale and a wide scale, that have to be defined. Some embodiments allowanalysts to specify scale values. Another approach is to assign thenarrow scale and a wide scale to specified multipliers, such as 5% and30% of the displayed map size. Accordingly, zooming automaticallyadjusts the narrow and wide scale to the display. Given narrow and widescales, local averages are made over rectangles, circles or otherregions at these scales. Some embodiments compensate for edge effects.

Some embodiments for real time applications, such as dynamic advertisingon mobile phones, use fast algorithms to access and analyze largedatasets with rapid retrieval. Some embodiments organize the data intopartitions. Some such embodiments store data in Hadoop in Parquet formatfor faster retrieval. Apache Parquet is a column-based storage formatfor Hadoop. Some datasets have many columns, but work with subsetsrather than entire records. Parquet optimizes this type of work.Further, Parquet can compress data using snappy compression oralternative compression techniques. This reduces storage spacerequirements and access latency. Apache Spark and Spark-SQL, withParquet, enable parallel and distributed access. Parquet, for example,can boost Spark SQL performance by 10 times compared to using text. Thisis due to the low-level reader filters, efficient execution plans, andgood scan throughput. Using Parquet with compression typically reducesdata storage by 75% compared to text.

Analyst Requests and Responses

In some embodiments, analysis is performed based on the mined data. Insome embodiments, analysts additionally mine data for special purposesbeyond the mining performed by data miners. FIG. 7 illustrates anembodiment that presents a uniform interface available to data minersand analysts. A data mining subsystem 101 provides access to a database102. The database 102 comprises a database of interpretations and theircontext 103. The database 102 also comprises a database of mined data104. In this integrated system, the distinction between data miners andanalysts is that data miners work for a data provider, and analysts arethe clients of the data provider. In some embodiments, the functionalityavailable to analysts is a subset of the functionality available to dataminers. The embodiment of FIG. 7 groups all data (the raw database andthe mined data) in a single joint database 102.

Analyzing data mined from the raw database, in various embodiments,involves lookups, filtering, to retrieve a subset of data, aggregatingaspects of the retrieved data, counting, matching, comparing, ranking,percentile calculation, building histograms, approximatingdistributions, displaying relationships of the aggregated data toregions, and other data processing functions. An example of a type ofanalysis is to count, for each region in a set of regions,interpretations or concepts with a specific property. One such propertyis the presence of a particular entity. For example, it might be usefulto count the number interpretations that include a fast food conceptduring a particular time period, such as between 5:00 pm and 7:00 pm.For analysis to be useful, databases must have a statisticallysignificant number of occurrences of expressions with the particularproperty.

Some embodiments enable analysts to filter database records according tothe presence of particular entities. Some embodiments allow analysts,additionally or alternatively, to filter results according to specificranges of values of particular attributes of particular entities. Forexample, it may be useful to retrieve concepts from interpretations ofexpressions that were collected within a specific location and timeperiod. In some embodiments, such filtering is based on matching morecomplex semantic relationships. For example, a query interpretation maybe constructed to find one or more interpretations that are similar tothe query interpretation for some measure of similarity. The degree ofmatch between two interpretations (such as a query interpretation and aderived/stored interpretation) is measured through fuzzy matching thatoccurs between the query interpretation and an interpretation stored inthe database, or the corresponding concepts. In embodiments whoseinterpretations are based on semantic graphs, the relative overlap oftwo semantic instances indicates a degree of match. In some embodiments,the similarity of query interpretations and derived interpretationsstored in a database are computed as a Jaccard index. In someembodiments, graph matching techniques are used to find overlappingsubgraphs. The maximally matching subgraph is determined, and a relativeoverlap factor is measured based on the size of the maximally overlappedgraph, and the ratio of its size relative to the database interpretationgraph's size. In all these similarity measures, weights are used toexpress the similarity of individual nodes or edges, and to give certainnode types or concepts a greater weight.

Some embodiments use the highest Jaccard index value to select thestored interpretation that best matches the query interpretation. Someembodiments choose a particular number of stored interpretations withthe highest index values. Some embodiments apply a fixed thresholdcoefficient value to determine a match set of stored interpretations.

Some embodiments, instead of or in addition to a Jaccard index,calculate match scores between query interpretations and interpretationsstored in database records using one or more of hamming distance, cosinesimilarity, and edit distance, among other conventional similarityscoring algorithms.

Some embodiments compute interpretation match scores based on presenceof entities. Some embodiments refine the scores by similarity ofattribute values. Some embodiments boost the score for entities withmatches between the presences of defined attributes.

Various embodiments provide various tools for analysts to define desiredanalysis and display of output. Some embodiments provide userinterfaces, like ones found in spreadsheet software, for users to createformulas for data processing. Some embodiments allow users to createcustom formulas. Some embodiments accept formulas programmed intext-based scripting or programming languages. Some embodiments allowfiltering and searching using regular expressions.

Analysts are generally interested in receiving selected informationabout the mined data. In some cases, already mined data is sufficientfor an analyst to find the answers the analyst seeks. In some suchcases, simple or more advanced display tools allow an analyst to explorethe relationships of interest.

Various embodiments allow analysts to use mined data to create a thoughtmap. A thought map shows the result of analysis of raw or mined datapositioned on a map of one or more geographical regions. A thought mapmay be embodied as a visual display, such as a computer screen, or avirtual reality display of a geographical heat map. Some embodimentsprovide a two-dimensional visual display showing multiple layers of dataoverlaid with a map. In some embodiments the map is scalable. Somedisplay tools can show time data. Some embodiments use auditorydisplays.

Some embodiments render geographical maps in browser windows, e.g.,using a language such as JavaScript. In some embodiments, the display ofinterpretations is a mashup with another geographical map rendering APIsuch as Google Maps.

Some embodiments use displays other than browser windows, such as customdesigned computer GUIs or paper printouts. Some embodiments provide aweb API for third party developers to access interpretation data.

In some embodiments, interpretations are displayed using words,geographically placed based on the center of their expression, butplaced based on a force-directed drawing algorithm. In some embodiments,the font size used to display interpretation-derived text depends on itsfrequency of use in expression interpretations, relative to the weightedfrequency of use of other expression interpretations. In someembodiments, the size of a text rectangle is based on a frequency, andthe font size is adjusted so that its rectangle is proportionallyscaled.

One type of heat map shows the emotional charge of expressions, whichcan be expressed as a grey transparency map overlay in which a dark greylevel conveys the negativity of the emotional charge of concepts in theexpressions. One type of heat map displays information about theemotional content of expressions. Some embodiments map emotional contentto a color scale and render it as a map overlay, such as, e.g.,joy=yellow, sadness=blue, disgust=green, fear=violet, anger=red, as inthe Pixar movie “Inside Out.”

Concepts implied, but not directly expressed in expressions, such as“the ball's in your court”, appear on the map. Directly-expressedconcepts appear if an analyst adjusts the level of concept strengththreshold in a filter. Some embodiments allow an analyst to listparticular concepts of interest, in which case general concept strengthis reduced in determining which to present, and concepts withsignificantly strong graph-connection weight to directly expressedconcepts are boosted in the display.

FIG. 10 illustrates a geomap with indications of concepts and locationswhere they were expressed according to an embodiment. FIG. 11 shows theconcept layer from FIG. 10, without the base geographical map. Someembodiments allow analysts to filter displayed results based on a timerange. Some embodiments allow filtering based on time of day. FIG. 12shows concepts expressed in the morning, FIG. 13 shows conceptsexpressed in the afternoon, and FIG. 14 shows concepts expressed in theevening.

Another type of thought map is a regional map with regions highlightedbased on the number of captured expressions with certain properties. Forexample, if an analyst applies a filter for expressions about food, themap will show the density of such expressions within each cell of a gridover the map region. Some embodiments display, for each location, theconcepts of greatest salience. Some embodiments display, for eachconcept, the location at which the salience is greatest. the expressionproperties of greatest salience near their location of highest salience.For example, if an analyst queries for the most common conceptsexpressed within a displayed map region, the map display may show wordsrepresenting the concepts most frequently expressed. In someembodiments, where displayable properties tend to overlap or becomecrowded, a visualization system makes only the properties or concepts ofgreater salience visible. When an analyst changes zoom level, the systemrecalculates heat map layers, densities, salience, and all visibilitydecisions.

Some embodiments display, for each location in which an expressedconcept matches a specified filter concept, a marker such as a dot isrendered. Some systems allow for zooming in and out, and merge dots thatare closer together than a particular level of resolution. Some systemsallow for placing a pointer, such as one controlled by a computer mouse,over an expression marker; in response to such hovering, such systemspop up a display that reveals the text of one or more expressionsindicated by the marker or reveal a selected aspect of theinterpretations. Some systems render markers using different colors andshapes based on specified entities or values of entity attributes andemotions in the expressions. Some systems overlay different types ofinformation, such as different colors, and text.

FIG. 8 shows an embodiment of map layers in an exemplary display. A baselayer 61 displays a geographical map, such as one obtained from GoogleMaps. A second layer 62 displays an emotion heat map as asemitransparent overlay over the first layer. A third layer 63 is anoverlay of dots showing points at which interpreted expressions havebeen captured. The display only displays points whose interpretationsand associated data match a specified filter, such as, for example,expressions gathered in the evening, whose concepts involve eating ordrinking, and that express questions about, or opinions about, places toeat or drink. The top layer 64 is an overlay of concept words chosen asrepresentative of the interpretation concepts.

Every layer in a display such as FIG. 8 is sensitive to scale. One clearreason for this is that the information density that can be packed intoeach display layer is limited by its legibility. It is useful andimportant, not only to limit the amount of data on a display layer, butto select carefully what data will be displayed, for a specific purposeand at scale. If the purpose provides a yes-no filter, only filtereddata will be considered. If the purpose provides a gradual measure ofrelevance, statistics (such as measures of popularity) will be weightedby relevance.

Thought maps do not need to represent locations according to terrestrialgeography. Some embodiments represent a thought map as a landmark graph.A landmark graph is a graph in which each node represents a landmark andeach edge represents a linear distance between the nodes it connects.Some embodiments display the graph visually, to scale, with edgesrepresenting greater distances drawn as longer than edges representingcloser distances. Some embodiments annotate the edges with a distancemeasure.

In an example graph, one node represents a particular big tree andanother connected node represents a particular stop sign that isnoteworthy to travelers because a vandal made a hole in it. The noderepresenting the stop sign with a hole connects to a node thatrepresents a particular vending machine.

Some embodiments can build another example graph from expressions madeby passengers giving directions to drivers, such as when traveling inthe town shown on the map in FIG. 9. One node represents a location inwhich the passenger said “turn onto Gold Street”. That node connects toa node that represents a location in which the passenger said “turn ontoCatherine street”. That node connects to a location in which thepassenger said “turn onto N Taylor Street”. That node connects to alocation in which the passenger said “turn onto Hoppe Street”. That nodeconnects to a location in which the passenger said “turn onto MoffatStreet”. Connecting the nodes, based on the order in which the passengermade the expressions makes a graph of driving action points. Graph edgesindicate the amount of time between each node, as derived from analyzingtime between different passengers' direction-giving expressions.

Representing locations by node/edge graphs can, for some applications,be even more useful than maps that represent locations geographically.For example, in an application for displaying advertisements on a mobilephone, it is important to know when a consumer, moving from node tonode, will likely soon arrive at a node near a particular place ofbusiness. Such a system would display an ad shortly before the consumerarrives. Using a strictly geographical map, to provide an advertisementwith such accurate timeliness, a system would have to know a trajectory,speed, and expected changes to speed and trajectory in order to estimatethe best time to display an ad.

Uses of the Data

In some embodiments, data analysts exploit the results found in themined data to achieve specific purposes. The raw and mined data storedin the databases may include more information than is needed for aparticular use of the data. Attributes of interpretations such aspersonal information (for example, the name and phone numbers ofparticular people), time, or emotional content can be ignored if notneeded.

In some embodiments, data analysts are agents of advertising companies,who seek to select the best location for billboards on which to place anadvertisement for a specific product or service.

Another use of the mined data is predicting user behavior. Someembodiments, shortly after capturing an expression, interpret theexpression; identify one or more identifiable behaviors; and store theinterpretation and the behavior as a record in a database. Someembodiments identify and store multiple behaviors in each databaserecord. Some embodiments capture movement behaviors, such as directionand magnitude of movement as detected by an accelerometer, or a changein location or speed as detected by a satellite geolocation system. Someembodiments capture purchasing behavior following expressions.

Some embodiments compare a recently received interpretation from aspecific user to previously stored interpretations for many users tocrowdsource predictions. By doing so, rather than predicting a userbehavior solely by the user's previous behavior, a system can predictthe user behavior from the behaviors of others following similarexpressions. Some embodiments store related user IDs with interpretationrecords in the database to associate expressions with particular users.Such embodiments filter interpretations from the database for comparisonin order to personalize predictions for individual users. Sinceindividual users have unique behavior patterns, by personalizingpredictions, rather than crowdsourcing predictions, a system can predictan individual's behavior more accurately if the system has personalizedbehavior history available.

Identifying sequences of interpretations is useful for analysis thatpredicts sequences of thoughts. Some embodiments store an interpretationderived from a recently captured expression with a link to theinterpretation derived from the previously captured expression, theinterpretation of the next captured expression, or both. Identifying anindividual person's sequences of thoughts allows prediction andcustomization of systems to provide assistance or advertising to theusers. For example, if a user, tends to expresses thoughts about cookingdinner shortly after expression thoughts about picking up children fromday care, then a system, when detecting an expression about day care,can predict that the user will soon think about cooking dinner.Identifying patterns of sequences of thoughts between people, such asones engaging in conversational dialog, provides for analysis of humanpsychological behavior, and the effects of particular expressions on thethoughts of listeners.

Incidentally, identifying sequences of interpretations is also usefulfor disambiguating expressions with multiple reasonable interpretations,such as, “Ed wanted to impress the girl with the pearl necklace”.

In another example, an electronic billboard detects the approach ofparticular people, and changes before they arrive. Such billboards inpublic places shared by many people detect commonality betweeninterpretations of each one's recent expressions, and choose ads basedon the common interest.

In another example, a public safety system detects interpretationsrelated to danger and strength of the interpretation and associatedemotion map. When multiple such expressions occur within closeproximity, based on the location and direction of movement of peoplemaking the expressions, the system identifies the likely source of thedanger. This is useful such as in case of a criminal shooting, aroundthe epicenter of an earthquake, or at serious car accidents.

For example, localized density spikes in the concept maps for conceptsrelated to car accidents, or slow traffic, can allow public safetysystems to identify incident locations before callers can describe alocation with sufficient accuracy for a response. Expressions aboutoffers to buy black market items or substances, such as certainnarcotics, indicate particular locations, and times of day that policepatrols can most effectively improve neighborhood safety.

Some embodiments are useful for epidemiology. Filtering interpretationsfor expressions explicitly indicating disease symptoms, or conceptsimplicitly related to disease symptoms allows mapping of density andspread of communicable illnesses. This can direct the distribution ofmedicines and travel patterns of doctors and directed application ofresources to improve sanitation or identify sources of environmentalhazards.

Time period filtering allows analysis of recurring disease spreadthroughout a year. Signal processing on changing density data indicatesper school zone of greatest weekly fluctuation. This indicates schoolsor workplaces where disease, such as influenza, spread is more frequent,and sanitation needs improvement.

A method of predicting disease outbreak, according to some embodiments,is to analyze historical expression interpretations in advance ofexplicit indications of disease symptoms.

Discarding personal identification information, and averaging individualinterpretations in advance of otherwise-detectable disease indicationallows the identification, and weighting, of predicted expressions.Though detecting such thoughts on an individual basis is a veryinaccurate predictor of impending health problems, detecting changingconcepts in expressions across large numbers of people within closegeographical proximity, and controlling for affecting current events,can indicate increased probability of an impending epidemic. Currentevents may have a very large influence on expressions. For example,airing of a television show about sharks will cause a widespreadincrease in all aquatic and marine related expressions. However, currentevents may cause high-frequency spikes within identifiable relationshipsbetween concept interpretations, whereas low-pass filtering changes infrequency of concepts is indicative of broader epidemiological trends inthe condition of the population.

Some embodiments that use landmark graphs (as described above) areuseful for navigation. Even if satellite constellation-based geolocationsystems, such as GPS, Galileo, GLONASS, and BeiDou are unavailable suchas for navigation in indoor locations such as underground shoppingmalls, jamming, or military attack, thought mapping can provide forprecise navigation. Such as system filters interpretations for onesspecifically about present locations. For example, “I am at SoundHound”would be included, but “I am going to The Computer History Museum” wouldbe ignored. The speaker indicates to the system a destination and locallandmarks in the current location. The system then instructs the user adirection for travel, and a several identifiable landmarks expectedalong the route. When the user requests the next step, the systemindicates a next set of landmarks. This is superior to simplelandmark-based navigation using data sources such as Google Maps becausesuch maps only have limited information, such as about the locations ofbusinesses and particular sites of interest. A large databases ofinterpretations has many fine-grained location-identifying clues from aspast expressions such as, “What a big tree”, “Look mom, somebody shot ahole in that stop sign”, and “This [vending] machine ate my dollar”.Children are particularly good sources of landmark data. Because oftheir tendency to verbalize what they learn about the world, theyprovide many expressions about notable landmarks about which few adultswould comment. By collecting expressions with geographical relevance, alandmark graph provides an association between expression and places,places being represented by particular objects.

Some embodiments are useful for investigating crimes and findingterrorists or rebels as they plot. A system takes in a set ofinterpretations from an investigator, matches them to concepts derivedfrom interpretations, and returns a list of locations at which they wereexpressed. Some embodiments take in sets of relevant interpretationvalues, such as connections to specific people, filter the database forexpressions containing that information, and return lists of locations.

Some such embodiments maintain a person entity graph with a node foreach identifiable speaking person and a directed edge to each otherperson who the speaker ever mentioned in an expression. Edges alsocomprise information about discernable relationships between people, aswell as frequency and recency of mentions. Person nodes accumulateattribute-value pairs over time as they can be determined from capturedexpressions. Similar person-graph embodiments may be useful foradvertisers.

Advertising Use Case and Example

The use of mined data may be applied to selecting a location in which toinvest resources. An example of investing resources in a location ischoosing the location of a billboard for advertising. Some embodimentsaccept ads from ad buyers and compute their relevance at any givenlocation and time. Some embodiments further consider the circumstancesof consumers in choosing ads to present. The ad buyers specify at leastconcepts and regions of interest in their bids. In some embodiments, adbuyers may express functions of concepts and region ranges. An exampleconcept function is whether expressions include the concept of Chinesefood. Another example of a concept function is if a weather concept isexpressed, does it contain concepts related to being hot, cold, orneither.

Various embodiments allow ad buyers to specify regions by one or more ofzip code, geopolitical definition, point of interest, latitude-longituderanges, or by custom locations boundaries, such as locations within viewof a billboard. Other methods of specifying filters are possible,including specifying phases within a time period, ranges of emotions,user attributes, and the like.

If a consumer makes an expression within the bid region of an ad, thenthe ad enters the bidding contest and ad bid functions run on theexpression's interpretation. Some embodiments select an advertisementbased on a fixed bid amount. Some embodiments take into account adistance weighting function, which affects an ad's bid amount inproportion to the distance of the expression's location to a centrallocation specified for the bids. In some embodiments, an ad's bid amountconsiders a conceptual distance function between the expression'sinterpretation and the concepts specified for the bids. Ad providers useproprietary formulas, any of which are appropriate for implementation ofembodiments of the present invention. Some embodiments use the highestbid amount to select an ad from the ad database for display.

Various embodiments present selected ads on billboards, display them onmobile devices such as a phones or automobiles, or play them as audiosegments on mobile or fixed devices. Any method of ad delivery may beused.

Some embodiments allow ad buyers to specify a bid amount as a functionof one or more of semantic information from expressions, environmentalinformation, and information provided in response to a request forinformation or an action. Some examples of environmental information aredemographic attributes, emotion, phase within a period, and one or morespecific person IDs.

Some embodiments present ad buyers with geographical insight intoexpression interpretations. FIG. 9-15 illustrates a simple scenario ofusing of such an embodiment. FIG. 9 shows a geographical map of a smalltown with Maria Elena's Mexican restaurant 801, Vahl's restaurant andbar 802, two bus stations 803 and 804, a US post office 805, and aGardner Family health clinic 806. The system captures queries spoken toa virtual assistant app on mobile phones. Over the course of a day, atMaria Elena's 801, a person asks, “Where should we take the kids afterlunch?”; at a bus station 804 a person asks, “When will the next busarrive?”; at Vahl's 802 a person asks, “What's in a Margarita?”; at thepost office 805 a person asks, “When will my package arrive?”; and atthe health clinic 806 a person asks, “Where can I fill a prescription?”.

FIG. 10 shows salient concepts accumulated throughout a given day, andgeographically, in the town. In and around Maria Elena's 801, kids,food, lunch, burritos, and cost were expressed prominently. On the shadyside of the block near a bus stop 803, concept words such as bus, cold,and wait were expressed in the morning; near the bus station 804 on thesunny side of the block, commute time, bus, and sun were expressed. AtVahl's restaurant 802, food, dinner, cocktails, and concept words in thesports domain were expressed. Throughout the evening, expressions weremade about different sports. Since every particular sport has a strongrelationship with the concept of sports, in general, sports gains ahigher weight than any individual sport. On a day with, for example, avery popular basketball game, this basketball concept might outweigh theconcept of sports, in general. Throughout the day, at the post office805, slow, junk mail, and packages were expressed. At the health clinic806, kids, sickness, doctor, medicine, and cost were expressed. Drivingconcepts were expressed on Gold street, heading out of town 807.Specifically, various concepts related to driving were expressedthroughout the day, and so the broader concept of driving appears withsignificant strength on the map. This effect, just like the sportsexample, depends on a hierarchy of concepts, represented taxonomicallyin the thought mapping system.

After analysis of the geographical interpretation map of FIG. 10, a beeradvertiser decides to bid on an advertisement on a billboard betweenMaria Elena's restaurant 801 and Vahl's bar 802, as shown in FIG. 15.The system analyzes expressions over the course of a month followingplacement of the billboard. A follow-up analysis shows that beer isexpressed at the bus stop 803, strongly at Maria Elena's restaurant 801,very strongly at Vahl's bar 802, and even by drivers on Gold Street 807.Capturing and interpreting expressions allows the advertiser to measurethe success of the particular ad, but also the ability of the billboardto influence people's thoughts. A billboard's influence depends on itsplacement, orientation, and its surrounding environment.

FIG. 16 is a block diagram of an example computer system. A computersystem 1610 typically includes at least one processor 1614, whichcommunicates with a number of peripheral devices via a bus subsystem1612. These peripheral devices may include a storage subsystem 1624,comprising for example memory devices and a file storage subsystem, userinterface input devices 1622, user interface output devices 1620, and anetwork interface subsystem 1616. The input and output devices allowuser interaction with the computer system 1610. A network interfacesubsystem 1616 provides an interface to outside networks and is coupledvia the communication network to corresponding interface devices inother computer systems.

User interface input devices 1622 may include a keyboard, pointingdevices such as a mouse, trackball, touchpad, or graphics tablet, ascanner, a touchscreen incorporated into the display, audio inputdevices such as voice recognition systems, microphones, and other typesof input devices. In general, use of the term “input device” is intendedto include all possible types of devices and ways to input informationinto a computer system 1610 or onto a communication network.

User interface output devices 1620 may include a display subsystem, aprinter, a fax machine, or non-visual displays such as audio outputdevices. The display subsystem may include a cathode ray tube (CRT), aflat-panel device such as a liquid crystal display (LCD), a projectiondevice, or some other mechanism for creating a visible image. Thedisplay subsystem may also provide non-visual display such as via audiooutput devices. In general, use of the term “output device” is intendedto include all possible types of devices and ways to output informationfrom computer system 1610 to the user or to another machine or computersystem.

The storage subsystem 1624 stores programming and data constructs thatprovide the functionality of some or all of the modules describedherein, including the logic to create inferred queries for use as querysuggestions according to the processes described herein. These softwaremodules are generally executed by a processor 1614 alone or incombination with other processors.

Memory 1626 used in the storage subsystem can include a number ofmemories including a main random access memory (RAM) 1630 for storage ofinstructions and data during program execution and a read only memory(ROM) 1632 in which fixed instructions are stored. A file storagesubsystem 1628 can provide persistent storage for program and datafiles, and may include a hard disk drive, a floppy disk drive along withassociated removable media, a CD-ROM drive, an optical drive, orremovable media cartridges. The modules implementing the functionalityof certain embodiments may be stored by the file storage subsystem 1628in the storage subsystem 1624, or in other machines accessible by theprocessor.

Bus subsystem 1612 provides a mechanism for letting the variouscomponents and subsystems of the computer system 1610 communicate witheach other as intended. Although bus subsystem 1612 is shownschematically as a single bus, alternative embodiments of the bussubsystem may use multiple busses.

The computer system 1610 can be of varying types including aworkstation, server, computing cluster, blade server, server farm, orany other data processing system or computing device. Due to theever-changing nature of computers and networks, the description of thecomputer system 1610 depicted in FIG. 16 is intended only as an examplefor purposes of illustrating the preferred embodiments. Many otherconfigurations of computer system 1610 are possible having more or fewercomponents than the computer system depicted in FIG. 16.

What is claimed is:
 1. A method of predicting a person's behavior, themethod comprising: storing, in a database, interpretations of priornatural language expressions obtained from a user device and one or moregeolocation based movement behaviors of the user device that occurredwithin a specific amount of time after the prior natural languageexpressions were received, each of the stored one or more geolocationbased movement behaviors being related to one or more of the storedinterpretations of the prior natural language expressions; receiving acurrent natural language expression after the one or more geolocationbased movement behaviors occurred; interpreting the current naturallanguage expression to create a current interpretation; comparing thecurrent interpretation to the stored interpretations of the priornatural language expressions to identify a closest matching storedinterpretation of the stored interpretations; predicting the person'sgeolocation based movement behavior by selecting a stored geolocationbased movement behavior, of the stored one or more geolocation basedmovement behaviors, related to the closest matching storedinterpretation; and providing the person's predicted geolocation basedmovement to an entity.
 2. The method of claim 1, further comprising:storing, in the database, user IDs in association with the storedinterpretations of the prior natural language expressions; and filteringthe database according to a current user ID of a user that uttered thecurrent natural language expression, wherein the filtering selects, fromamong the stored interpretations, interpretations that are associatedwith user IDs that match the current user ID.
 3. The method of claim 1,wherein the prior natural language expressions are unrelated to anoperational function of the user device.
 4. At least one non-transitorycomputer readable medium storing code that, if executed by at least onecomputer processor would cause the at least one computer processor to:store, in a database, interpretations of prior natural languageexpressions obtained from a user device and one or more geolocationbased movement behaviors of the user device that occurred within aspecific amount of time after the prior natural language expressionswere received, each of the stored one or more geolocation based movementbehaviors being related to one or more of the stored interpretations ofthe prior natural language expressions; receive a current naturallanguage expression after the one or more geolocation based movementbehaviors occurred; interpret the current natural language expression tocreate a current interpretation; compare the current interpretation tothe stored interpretations of the prior natural language expressions toidentify a closest matching stored interpretation of the storedinterpretations; predict the person's geolocation based movementbehavior by selecting a stored geolocation based movement behavior, ofthe stored one or more geolocation based movement behaviors, related tothe closest matching stored interpretation; and provide the person'spredicted geolocation based movement to an entity.
 5. The at least onenon-transitory computer readable medium of claim 4, wherein the code, ifexecuted by at least one computer processor further causes the at leastone computer processor to: store, in the database, user IDs inassociation with the stored interpretations of the prior naturallanguage expressions; and filter the database according to a currentuser ID of a user that uttered the current natural language expression,by selecting, from among the stored interpretations, interpretationsthat are associated with user IDs that match the current user ID.
 6. Theat least one non-transitory computer readable medium of claim 4, whereinthe prior natural language expressions are unrelated to an operationalfunction of the user device.
 7. A system including one or moreprocessors coupled to memory, the memory loaded with computerinstructions to predict a person's behavior, the instructions, whenexecuted on the processors, implement actions comprising: storing, in adatabase, interpretations of prior natural language expressions obtainedfrom a user device and one or more geolocation based movement behaviorsof the user device that occurred within a specific amount of time afterthe prior natural language expressions were received, each of the storedone or more geolocation based movement behaviors being related to one ormore of the stored interpretations of the prior natural languageexpressions; receiving a current natural language expression after theone or more geolocation based movement behaviors occurred; interpretingthe current natural language expression to create a currentinterpretation; comparing the current interpretation to the storedinterpretations of the prior natural language expressions to identify aclosest matching stored interpretation of the stored interpretations;predicting the person's geolocation based movement behavior by selectinga stored geolocation based movement behavior, of the stored one or moregeolocation based movement behaviors, related to the closest matchingstored interpretation; and providing the person's predicted geolocationbased movement to an entity.